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ABSTRACT 



This paper describes an evaluation strategy designed to 
assess the efficacy of an education system at the district level. The 
strategy provides a measure of accountability that takes into account the 
potential effects of demographic variables. The study focused on Bellevue 
Public Schools (Washington) , a medium-sized district near Seattle with an 
enrollment of 15,000 students. National, regional, and district data files 
were merged to form a database for aggregation and analysis. A correlational 
analysis was performed to identify demographic variables related to student 
achievement measures. Then, variables shown to be related to student 
achievement were included in a multiple regression analysis to assess the 
extent to which these demographic variables affect achievement and to 
construct a composite index of their impact. Third, the composite index was 
used to cluster school districts that were demographically similar to provide 
a basis for comparison. Variables shown to be related to student achievement 
at the district level included percentages of students from ethnic 
minorities, in special education, below the poverty level, in bilingual 
programs, or in compensatory reading programs. When the Bellevue district was 
compared with other Washington districts it most closely resembled, students 
in Bellevue consistently showed a higher level of achievement than the state 
average. This evaluation strategy provides a way of assessing the efficacy of 
the education system at the district level. It provides a measure of 
accountability that is more comprehensive and systemic than the evaluation of 
an isolated or categorical program. (Contains five tables and four 
references.) (SLD) 
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System Performance at the District Level: Demographics and Student 

Achievement 

Kim O. Yap 

Northwest Regional Educational Laboratory 



Introduction 

Evaluations have often been conducted at the program level without regard to the fact 
that programs implemented by a school or school district invariably interact with one 
another and that students are often participants in several district-, state-, or federally 
funded programs. It is often difficult, if not impossible, to partial out the interactive 
effects among programs when one attempts to assess the impact of each program 
separately. 

More recently, the educational research and development community has paid 
increased attention to the benefits of program and service integration (e.g., Melaville & 
Blank, 1991, Winfield, et al., 1992), advocating, for example, integrated curriculum 
and assessment. Likewise, federal education legislation (e.g., Improving America's 
Schools Act) has placed increased emphasis on the coordination and integration of 
categorical programs to help students perform to higher standards (U. S. Department of 
Education & American Educational Research Association, 1995). Service integration is 
to attend, simultaneously, to the health, education, and social needs of students. 

In the context of multiple, interactive programs implemented within a single system 
(e.g., the school district), evaluation models designed to assess impact at the program 
level are no longer adequate, or appropriate, for addressing accountability issues at the 
system level. Increasingly, there is a need to use broader indicators of efficacy to 
assess system impact at the district level. 

The efficacy of a district is influenced by a wide array of variables. Some of the 
variables are manipulable (e.g., curriculum and instruction, staff development), others 
are beyond the district's control. Among the uncontrolled variables are the backgrounds 
and characteristics of students who enroll in the school system. For example, there is 
abundant evidence (U.S. Department of Education, 1993) that poverty is a potent factor 
affecting the achievement of not only poor children but others who attend the same 
school. A system of assessment and accountability that is fair and meaningful should 
include a mechanism for identifying and partialling out the effects of uncontrolled 
demographic variables in assessing impact. 

This paper describes an evaluation strategy designed to assess the efficacy of an 
education system at the district level. The strategy provides a measure of accountability 
which takes into account the potential effects of demographic variables. The study was 
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focused on Bellevue Public Schools in Washington, a medium-sized district near 
Seattle with an enrollment of 15,000 students. 



Methods and Data Source 

The state of Washington has approximately 300 school districts offering a wide variety of 
instructional programs to students. The student population is diverse and the districts 
differ widely with respect to student ethnicity, poverty level, and other demographic 
characteristics. 

To study the effects of demographics on student achievement at the district level, the 
relevant data were extracted from three existing databases. Specifically, data on ethnicity 
and special education status were obtained from the Common Core Data (CCD) compiled 
by the National Center for Education Statistics (NCES). Poverty data were extracted 
from the 1990 census data. Data on student achievement and participation in special 
programs were drawn from a statewide database maintained by the Washington Office of 
the Superintendent of Public Instruction (OSPI) as well as data files and documents 
maintained by school districts. 

Norm-referenced tests provide the most widely used measure of students’ achievement in 
Washington. For several years, the state has used the Comprehensive Test of Basic Skills 
(CTBS/4) to assess student performance at grades four and eight, and the Curriculum 
Frameworks Assessment System (CFAS) at grade 1 1 . These tests provide a basis for 
comparing student achievement with a national (CTBS) or a state (CFAS) norm group, 
and the test results can reveal the relative strengths and weaknesses of instructional 
programs. Longitudinal results can also show patterns or trends in student performance 
across school years. 

Student achievement was measured by normal curve equivalent (NCE) scores. Reading 
NCE scores for grades four and eight were based on CTBS. English NCE scores for 
grade 1 1 were based on CFAS. The NCE scale, an equal-interval scale with a range of 
1 to 99, coincides with the percentile scale at the 1st percentile, 50th percentile, and 99th 
percentile. 



Data Analysis 

The national, regional, state, and district data files were merged to form a database for 
aggregation and analysis. Using districts as the unit of analysis, three sets of analyses 
were conducted on the data. First, a correlational analysis was performed to identify 
demographic variables related to the student achievement measures. Second, variables 
shown to be related to student achievement were included in a multiple regression 
analysis to assess the extent to which these demographic variables are affecting student 
achievement and to construct a composite index of their impact. Third, the composite 
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index was used to cluster school districts that are demographically similar to provide a 
basis for comparing districts with respect to student achievement. Because of missing 
data, a small number of districts were excluded from the analysis. 



Results 

The correlational analysis showed that several demographic variables were related to 
student achievement at the district level, including: 

• Percent of ethnic minority students 

• Percent of special education students 

• Percent of students below poverty level 

• Percent of students in bilingual programs 

• Percent of students in compensatory reading programs 

The relationships with percent of ethnic minority students and percent of students below 
poverty level were particularly strong, with the correlation coefficients hovering around 
.50, as shown in Table 1 . 
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Table 1 



Correlations Between Reading/English NCE Scores and Demographic Variables 



Demographic Variable 


Grade 

Level 


Correlation 

Coefficient 


% of ethnic minority students 


4 


-.54** 




8 


-.52** 




11 


_ 43 ** 


% of special education students 


4 


-.11 




8 


-.16 




11 


-.17* 


% of students below poverty level 


4 


-.50** 




8 


_ 47 * * 




11 


- 43* * 


% of students in bilingual programs 


4 


-.35** 




8 


- 25 ** 




11 


-.04 


% of students in compensatory reading 


4 


_ 32** 


programs 


8 


-.19* 




11 


N.A. 



* p < .05 

** p < .01 

Note. 1. Percent of students below poverty level was based on 1990 census data. 

2. Percent of ethnic minority students and percent of special education students 
were extracted from the 1 992-93 Common Core Data (CCD) compiled by the 
National Center for Education Statistics (NCES). 

3. Percent of students in bilingual programs and percent of students in 
compensatory reading programs came from a database maintained by the 
Washington Office of the Superintendent of Public Instruction (OSPI). 

4. The poverty thresholds are revised annually to allow for changes in the cost of 
living as reflected in the Consumer Price Index. The average poverty threshold 
for a family of four persons was $12,674 in 1989. Poverty thresholds were 
applied on a national basis and were not adjusted for regional, state, or local 
variations in the cost living. 

5. Reading/English NCE scores are for the 1990-91 school year. 
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Using the demographic variables as independent variables, the multiple regression 
analysis yielded multiple correlations of .62, .59, and .52 for grades 4, 8, 11, respectively. 
All correlations are significant beyond the .001 level. Table 2 provides a summary of the 
results. 



Table 2 

Multiple Correlations Between Dependent and Independent Variables 



Grad 

e 

Level 


N 


Independent 

Variables 


Dependent 

Variables 


Multiple 

R 


R 

Square 


P 


4 


284 


(1) % of ethnic 
minority students 

(2) % of special 
education students 

(3) % of students 
below poverty level 

(4) % of students in 
bilingual programs 

(5) % of students in 
compensatory 
reading programs 


CTBS 
reading 
NCE score 


.62 


.38 


<.001 


8 


261 


(1) % of ethnic 
minority students 

(2) % of special 
education students 

(3) % of students 
below poverty level 

(4) % of students in 
bilingual programs 

(5) % of students in 
compensatory 
reading programs 


CTBS 
reading 
NCE score 


.59 


.35 


<.001 


11 


238 


(1) % of ethnic 
minority students 

(2) % of special 
education students 

(3) % of students 
below poverty level 

(4) % of students in 
bilingual programs 


CFAS 
English 
NCE Score 


.52 


.27 


<.001 
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Results of the regression analysis were used to create a composite demographic index for 
each of the districts included in the analysis. The regression equation for each grade level 
included in the study provides an estimate of achievement that is solely determined by the 
demographic variables. In this study, this predicted achievement level served as the 
composite demographic index. It was used to rank and cluster districts with respect to the 
demographic variables. 

Then, using the composite index, a cluster of districts which most closely resembled the 
Bellevue School District with respect to the demographic variables were identified as 
comparison districts. For each grade level, two clusters were identified. The first cluster 
included four districts. The second cluster consisted of ten districts. 

Student Achievement 

Students in Bellevue consistently showed a higher level of achievement than the state 
average. During a four-year period from 1990 to 1994, the district scored higher on the 
CTBS and the CFAS. In most cases, the district’s average score was seven or more 
points higher than the state average. This is one-third or more of a standard deviation of 
the NCE scores. Such differences are generally considered to be educationally 
significant. 

With its high level of achievement, the district ranked very favorably in comparison with 
other districts in the state. During the four-year period, Bellevue scored in the top 
quartile of the state for all three grade levels included in the study. The overall 
achievement data are summarized in Table 3. 




6 



Table 3 



NCE Scores and District Ranks in Reading/English 



School Year 


90-91 


91-92 


92-93 


93-94 


Grade 4 










State Average 


50.5 


49.2 


48.4 


48.2 


Bellevue Average 


57 


57 


55 


57 


Bellevue Rank 


38 


25 


39 


26 


Number of Districts Included 


284 


284 


284 


284 


Grade 8 










State Average 


52.9 


52.8 


52.2 


52.3 


Bellevue Average 


60 


59 


60 


60 


Bellevue Rank 


29 


29 


23 


20 


Number of Districts Included 


261 


261 


261 


261 


Grade 11 










State Average 


48.2 


48.2 


47.3 


47.8 


Bellevue Average 


57 


57 


56 


55 


Bellevue Rank 


11 


15 


12 


14 


Number of Districts Included 


238 


238 


238 


238 



District Comparison 

When Bellevue was compared with districts which most closely resemble the district with 
respect to the demographic variables used in the study, its achievement level was, again, 
shown to be higher than that of the comparison districts. 

The evaluation methodology first identified demographic variables which are related to 
student achievement as measured by the CTBS and the CFAS. These variables included: 

• Percent of ethnic minority students 

• Percent of special education students 

• Percent of students below poverty level 

• Percent of students in bilingual programs 

• Percent of students in compensatory reading programs 
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There were some significant differences between Bellevue and the state as a whole with 
respect to these demographic variables. For example, the data showed that the district 
had a much higher percentage of ethnic minority students (26%) in comparison with the 
state average (17%). On the other hand, the data showed that 7 percent of the district’s 
students lived below the poverty level when the state average was more than 1 8 percent. 
Overall, the district had a higher percentage of students participating in bilingual 
programs and a much lower percentage of students in compensatory reading programs in 
comparison with the state average. The demographic data are summarized in Table 4. 



Table 4 

Summary Statistics on Demographic Variables 



Variable 


Grade Level 


Bellevue 


State 


% of ethnic minority 
students 


All grades 


26.0 


17.0 


% of special education 
students 


All grades 


9.0 


10.1 


% of students below 
poverty level 


All grades 


7.0 


18.3 


% of students in bilingual 


Grade 4 


1.2 


1.3 


programs 


Grade 8 


1.2 


.6 




Grade 1 1 


2.3 


.4 


% of students in 


Grade 4 


.4 


10.1 


compensatory reading 
programs 


Grade 8 


0 


3.3 



Note. 1 . Percent of students below poverty level was based on 1990 census data. 

2. Percent of ethnic minority students and percent of special education students 
were extracted from the 1 992-93 Common Core Data (CCD) compiled by the 
National Center for Education Statistics (NCES). 

3. Percent of students in bilingual programs and percent of students in 
compensatory reading programs came from a database maintained by the 
Washington Office of the Superintendent of Public Instruction (OSPI). 

4. The statewide statistics were based on 284 districts for grade 4; 261 districts for 
grade 8; and 238 districts for grade 1 1 . 



The methodology then put districts that are similar with respect to these variables in 
clusters. Bellevue was then compared with the similar districts. The results of these 
comparisons are summarized in Table 5. 
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The data showed that when Bellevue was compared with four demographically similar 
districts, its achievement level was consistently above those of the comparison districts 
for all four school years included in the study. When the district was compared with a 
larger cluster of ten demographically similar districts, the differences were, again, very 
favorable to the district. 



Table 5 



Reading/English NCE Scores of Bellevue and Comparison Districts 



Grade 

Level 


School 

Year 


Bellevue 


1st Cluster of 
Comparison 
Districts 


2nd Cluster of 
Comparison 
Districts 


4 


90-91 


57 


50.5 


51.7 




91-92 


57 


47.5 


49.2 




92-93 


55 


48.5 


49.1 




93-94 


57 


47.25 


48.1 


8 


90-91 


60 


59.25 


56.7 




91-92 


59 


55 


52 




92-93 


60 


56.5 


53 




93-94 


60 


58.75 


55.3 


11 


90-91 


57 


47.75 


49.4 




91-92 


57 


50 


50.4 




92-93 


56 


49.5 


49.6 




93-94 


55 


50 


50.3 



Note. 1 . Reading NCE scores for grades 4 and 8 were based on CTBS. 

2. English NCE scores for grade 1 1 were based on CFAS. 

3. The first cluster consisted of four districts which most closely resemble 

Bellevue with respect to the demographic variables used in the study. 

4. The second cluster consisted of ten districts which most closely resemble 
Bellevue with respect to the demographic variables used in the study. 
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Concluding Remarks 



The evaluation strategy described in this paper provides a way of assessing the efficacy 
of the education system at the district level. It provides a measure of accountability that 
is more comprehensive and systemic than the evaluation of an isolated or categorical 
program. In taking into account the potential effects of student demographics, it 
presents policymakers with an accountability tool that is fair and perhaps more 
acceptable to school and district administrators. 

The evaluation strategy is particularly pertinent under the new Title I legislation which 
strongly encourages schoolwide programs, program integration, and district level 
accountability. To be fair and equitable, it is important that the process for determining 
adequate yearly progress toward meeting improvement goals takes into account all 
critical variables that affect student performance — including those that are beyond the 
control of teachers and school administrators. 
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